OT Futility Analysis

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

OT Futility Analysis

Kevin E. Thorpe
I beg your pardon if this is too off topic.  I am posting here
since I hope to find an R solution to my problem.  Please indulge
me while I give a little background about what I'm trying to do.

I'm on a DSMB for a clinical trial.  The Steering Committee for the
trial has asked us to perform a futility analysis on their primary
outcome which is a time-to-event endpoint.  The trial was not designed
with group sequential methods, nor was any futility analysis spelled
out in the protocol.  Another thing which may be relevant is that
due to circumstances beyond the investigators' control, the trial
will stop recruitment prematurely unless there is some compelling
reason for them to find a way to continue the trial.  Lastly, the
trial has accrued not quite half of the planned sample size.

Admittedly, I don't have a vast amount of experience implementing
stopping rules.  In other protocols I have seen where futility
analyses have been planned but a group sequential design has not
otherwise been employed, conditional power has been used for the
futility rule.  So naturally, that was my first thought (although
I may well be wrong) in this case.  I have done RSiteSearch() with
the following terms (three different searches):

        futility analysis
        conditional power
        stochastic curtailment

Nothing that looked relevant to my problem jumped out at me.

I have read, somewhat recently, that there are problems with conditional
power, although I don't remember the details at the moment.  This
has prompted me to consider other approaches to the problem.

One simple thing that has occurred to me, although I don't know
what the implications are is to simply look at a confidence
interval around the hazard ratio for the treatment effect.  In
the event that the CI includes 1 and excludes any clinically
important difference, I would take that as an indication of
futility.

I would appreciate your comments on this and to learn of any more
formal methods, particularly of implementations in R.

Thank you for reading.

Kevin

--
Kevin E. Thorpe
Biostatistician/Trialist, Knowledge Translation Program
Assistant Professor, Department of Public Health Sciences
Faculty of Medicine, University of Toronto
email: [hidden email]  Tel: 416.946.8081  Fax: 416.946.3297

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: OT Futility Analysis

Spencer Graves
          What does this particular Steering Committee think a "futility
analysis" is?  Do they have any particular reference(s)?  What do you
find in your own literature review?

          If it were my problem, I think I'd start with questions like that.
Your comments suggested to me a confounding of technical and political
problems.  The politics suggests the language you need to use in your
response.  Beyond that, I've never heard before of a "futility
analysis", but I think I could do one by just trying to be clear about
the options the Steering Committee might consider plausible and then
comparing them with appropriate simulations -- summarized as confidence
intervals, as you suggest.

          And I hope that someone else will enlighten us both if there are
better options available.

          Best Wishes,
          spencer graves
p.s.  For any attorneys who may read these comments, the suggestions are
obviously warranteed up to the amount you paid for it, which is nothing.
  If you follow them and they turn out to be inappropriate, you will pay
the price.  I encourage you to share the problems with me, so I can
learn from the experience.  However, the limits of my liability are as
already stated.

Kevin E. Thorpe wrote:

> I beg your pardon if this is too off topic.  I am posting here
> since I hope to find an R solution to my problem.  Please indulge
> me while I give a little background about what I'm trying to do.
>
> I'm on a DSMB for a clinical trial.  The Steering Committee for the
> trial has asked us to perform a futility analysis on their primary
> outcome which is a time-to-event endpoint.  The trial was not designed
> with group sequential methods, nor was any futility analysis spelled
> out in the protocol.  Another thing which may be relevant is that
> due to circumstances beyond the investigators' control, the trial
> will stop recruitment prematurely unless there is some compelling
> reason for them to find a way to continue the trial.  Lastly, the
> trial has accrued not quite half of the planned sample size.
>
> Admittedly, I don't have a vast amount of experience implementing
> stopping rules.  In other protocols I have seen where futility
> analyses have been planned but a group sequential design has not
> otherwise been employed, conditional power has been used for the
> futility rule.  So naturally, that was my first thought (although
> I may well be wrong) in this case.  I have done RSiteSearch() with
> the following terms (three different searches):
>
> futility analysis
> conditional power
> stochastic curtailment
>
> Nothing that looked relevant to my problem jumped out at me.
>
> I have read, somewhat recently, that there are problems with conditional
> power, although I don't remember the details at the moment.  This
> has prompted me to consider other approaches to the problem.
>
> One simple thing that has occurred to me, although I don't know
> what the implications are is to simply look at a confidence
> interval around the hazard ratio for the treatment effect.  In
> the event that the CI includes 1 and excludes any clinically
> important difference, I would take that as an indication of
> futility.
>
> I would appreciate your comments on this and to learn of any more
> formal methods, particularly of implementations in R.
>
> Thank you for reading.
>
> Kevin
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: OT Futility Analysis

Stephen A Roberts

I would take the line that if they hadn't pre-specified any stopping rules, the only reason to stop is safety or new external data. I would be very suspicious of requests from the steering committee to stop for futility - they should be blinded so why are they thinking futility unless results have leaked? I would argue that they are obliged to finish the trial once they start.

This is an example of the need to sort out these things in advance - look up the stuff from the UK DAMOCLES project. The recent book edited by DeMets et al (Data Monitoring in Clinical Trials: A Case Studies Approach) is a good read on these sorts of issues and I think there is a more statistical book from the same group of authors.

 As far as software is concerned, futility analysis and conditional power are simply standard analyses with made up data and more-or-less justifiable assumptions.

Steve.


> -----Original Message-----
> From: [hidden email] [mailto:r-help-
> [hidden email]] On Behalf Of Spencer Graves
> Sent: 22 February 2006 03:45
> To: Kevin E. Thorpe
> Cc: R Help Mailing List
> Subject: Re: [R] OT Futility Analysis
>
>  What does this particular Steering Committee think a "futility
> analysis" is?  Do they have any particular reference(s)?  What do you
> find in your own literature review?
>
>  If it were my problem, I think I'd start with questions like that.
> Your comments suggested to me a confounding of technical and political
> problems.  The politics suggests the language you need to use in your
> response.  Beyond that, I've never heard before of a "futility
> analysis", but I think I could do one by just trying to be clear about
> the options the Steering Committee might consider plausible and then
> comparing them with appropriate simulations -- summarized as confidence
> intervals, as you suggest.
>
>  And I hope that someone else will enlighten us both if there are
> better options available.
>
>  Best Wishes,
>  spencer graves
> p.s.  For any attorneys who may read these comments, the suggestions are
> obviously warranteed up to the amount you paid for it, which is nothing.
>   If you follow them and they turn out to be inappropriate, you will pay
> the price.  I encourage you to share the problems with me, so I can
> learn from the experience.  However, the limits of my liability are as
> already stated.
>
> Kevin E. Thorpe wrote:
>
> > I beg your pardon if this is too off topic.  I am posting here
> > since I hope to find an R solution to my problem.  Please indulge
> > me while I give a little background about what I'm trying to do.
> >
> > I'm on a DSMB for a clinical trial.  The Steering Committee for the
> > trial has asked us to perform a futility analysis on their primary
> > outcome which is a time-to-event endpoint.  The trial was not designed
> > with group sequential methods, nor was any futility analysis spelled
> > out in the protocol.  Another thing which may be relevant is that
> > due to circumstances beyond the investigators' control, the trial
> > will stop recruitment prematurely unless there is some compelling
> > reason for them to find a way to continue the trial.  Lastly, the
> > trial has accrued not quite half of the planned sample size.
> >
> > Admittedly, I don't have a vast amount of experience implementing
> > stopping rules.  In other protocols I have seen where futility
> > analyses have been planned but a group sequential design has not
> > otherwise been employed, conditional power has been used for the
> > futility rule.  So naturally, that was my first thought (although
> > I may well be wrong) in this case.  I have done RSiteSearch() with
> > the following terms (three different searches):
> >
> > futility analysis
> > conditional power
> > stochastic curtailment
> >
> > Nothing that looked relevant to my problem jumped out at me.
> >
> > I have read, somewhat recently, that there are problems with conditional
> > power, although I don't remember the details at the moment.  This
> > has prompted me to consider other approaches to the problem.
> >
> > One simple thing that has occurred to me, although I don't know
> > what the implications are is to simply look at a confidence
> > interval around the hazard ratio for the treatment effect.  In
> > the event that the CI includes 1 and excludes any clinically
> > important difference, I would take that as an indication of
> > futility.
> >
> > I would appreciate your comments on this and to learn of any more
> > formal methods, particularly of implementations in R.
> >
> > Thank you for reading.
> >
> > Kevin
> >
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-
> guide.html

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: OT Futility Analysis

Kevin E. Thorpe
Thank you Spencer and Steve for your helpful comments.  If I may, I
would like to elaborate on some of the points you raise.

Stephen A Roberts wrote:
> I would take the line that if they hadn't pre-specified any stopping
> rules, the only reason to stop is safety or new external data. I
> would be very suspicious of requests from the steering committee to
> stop for futility - they should be blinded so why are they thinking
> futility unless results have leaked? I would argue that they are
> obliged to finish the trial once they start.

In general I agree with this.  In this case the request for a futility
analysis came from the sponsor (a drug company).  It is a classic case
of company B buys company A and wnats to stop R&D on company A's drugs.
Therefore the company was looking for a reason to stop.  Now that they
will stop producing the drug used in the trial, recruitment will end
before reaching its target.  Now the Steering Committee's point of
view is that if there is any reasonable hope, they would find some
other way to continue recruitment.  I am confident that results have
not leaked.  I am well aquainted with the data management and blinding
procedures in place for the trial.

> This is an example of the need to sort out these things in advance -
> look up the stuff from the UK DAMOCLES project. The recent book
> edited by DeMets et al (Data Monitoring in Clinical Trials: A Case
> Studies Approach) is a good read on these sorts of issues and I think
> there is a more statistical book from the same group of authors.

Thanks for the reference.  My library has it, so will give it a look.

> As far as software is concerned, futility analysis and conditional
> power are simply standard analyses with made up data and more-or-less
> justifiable assumptions.

I am also interested if there are good alternatives to conditional
power for this type of scenario.

> Steve.
>
>
>
>>
>> What does this particular Steering Committee think a "futility
>> analysis" is?  Do they have any particular reference(s)?  What do
>> you find in your own literature review?
>>
>> If it were my problem, I think I'd start with questions like that.
>> Your comments suggested to me a confounding of technical and
>> political problems.  The politics suggests the language you need to
>> use in your response.  Beyond that, I've never heard before of a
>> "futility analysis", but I think I could do one by just trying to
>> be clear about the options the Steering Committee might consider
>> plausible and then comparing them with appropriate simulations --
>> summarized as confidence intervals, as you suggest.

I did ask REPEATEDLY for guidelines from the steering committee, but
none came or are likely to come.  In fact, they wanted me to come up
with the recommendation, which I find entirely inappropriate, but here
I am.  So, I don't think I'm confounded between techincal and political.

Basically, they want to stop if there is a low chance of rejecting the
null hypothesis.  This is often referred to as conditional power or
stochastic curtailment.  I recently saw a paper by Scott Emerson
pointing out some problems (interpretation, relation to unconditional
power).

As far as references, I have used a book by Jennisen and Turnbull in
the past, but, as I recall, with the exception of stochastic
curtailment, it assumes the trial was designed with group sequential
methods.  I have also just found a 1988 Biometrics paper by Lan and
Wittes on the B-value which I will read.

>> And I hope that someone else will enlighten us both if there are
>> better options available.
>>
>> Best Wishes, spencer graves p.s.  For any attorneys who may read
>> these comments, the suggestions are obviously warranteed up to the
>> amount you paid for it, which is nothing. If you follow them and
>> they turn out to be inappropriate, you will pay the price.  I
>> encourage you to share the problems with me, so I can learn from
>> the experience.  However, the limits of my liability are as already
>> stated.
>>
>> Kevin E. Thorpe wrote:
>>
>>
>>> I beg your pardon if this is too off topic.  I am posting here
>>> since I hope to find an R solution to my problem.  Please indulge
>>>  me while I give a little background about what I'm trying to do.
>>>
>>>
>>> I'm on a DSMB for a clinical trial.  The Steering Committee for
>>> the trial has asked us to perform a futility analysis on their
>>> primary outcome which is a time-to-event endpoint.  The trial was
>>> not designed with group sequential methods, nor was any futility
>>> analysis spelled out in the protocol.  Another thing which may be
>>> relevant is that due to circumstances beyond the investigators'
>>> control, the trial will stop recruitment prematurely unless there
>>> is some compelling reason for them to find a way to continue the
>>> trial.  Lastly, the trial has accrued not quite half of the
>>> planned sample size.
>>>
>>> Admittedly, I don't have a vast amount of experience implementing
>>>  stopping rules.  In other protocols I have seen where futility
>>> analyses have been planned but a group sequential design has not
>>> otherwise been employed, conditional power has been used for the
>>> futility rule.  So naturally, that was my first thought (although
>>>  I may well be wrong) in this case.  I have done RSiteSearch()
>>> with the following terms (three different searches):
>>>
>>> futility analysis conditional power stochastic curtailment
>>>
>>> Nothing that looked relevant to my problem jumped out at me.
>>>
>>> I have read, somewhat recently, that there are problems with
>>> conditional power, although I don't remember the details at the
>>> moment.  This has prompted me to consider other approaches to the
>>> problem.
>>>
>>> One simple thing that has occurred to me, although I don't know
>>> what the implications are is to simply look at a confidence
>>> interval around the hazard ratio for the treatment effect.  In
>>> the event that the CI includes 1 and excludes any clinically
>>> important difference, I would take that as an indication of
>>> futility.
>>>
>>> I would appreciate your comments on this and to learn of any more
>>>  formal methods, particularly of implementations in R.
>>>
>>> Thank you for reading.
>>>
>>> Kevin


--
Kevin E. Thorpe
Biostatistician/Trialist, Knowledge Translation Program
Assistant Professor, Department of Public Health Sciences
Faculty of Medicine, University of Toronto
email: [hidden email]  Tel: 416.946.8081  Fax: 416.946.3297

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: OT Futility Analysis

Marc Schwartz (via MN)
On Wed, 2006-02-22 at 11:57 -0500, Kevin E. Thorpe wrote:
> Thank you Spencer and Steve for your helpful comments.  If I may, I
> would like to elaborate on some of the points you raise.

Kevin,

I am not sure if you received any offlist replies to your post. Given
the subject matter, I had considered that you might have.

You might find the following thread over in the MedStats group to be of
interest:

http://groups.google.com/group/MedStats/browse_frm/thread/144c97dc5cfc4f00?tvc=1

It discusses some of the issues of early stopping, in this particular
case due to "running out of funds". Some of the points raised below are
addressed in that thread.

MedStats, BTW, would be a good forum to consider for your query.

> Stephen A Roberts wrote:
> > I would take the line that if they hadn't pre-specified any stopping
> > rules, the only reason to stop is safety or new external data. I
> > would be very suspicious of requests from the steering committee to
> > stop for futility - they should be blinded so why are they thinking
> > futility unless results have leaked? I would argue that they are
> > obliged to finish the trial once they start.
>
> In general I agree with this.  In this case the request for a futility
> analysis came from the sponsor (a drug company).  It is a classic case
> of company B buys company A and wnats to stop R&D on company A's drugs.
> Therefore the company was looking for a reason to stop.  Now that they
> will stop producing the drug used in the trial, recruitment will end
> before reaching its target.  Now the Steering Committee's point of
> view is that if there is any reasonable hope, they would find some
> other way to continue recruitment.  I am confident that results have
> not leaked.  I am well aquainted with the data management and blinding
> procedures in place for the trial.

Has the decision to cease production already been made or is the sponsor
still open to being "sold" on the idea of keeping the study going,
pending the outcome of your further work?

If production of the study treatment has already ceased, the ability of
the SC to make a business case to the sponsor may be a forgone
conclusion if there is insufficient product available to continue.

> > This is an example of the need to sort out these things in advance -
> > look up the stuff from the UK DAMOCLES project. The recent book
> > edited by DeMets et al (Data Monitoring in Clinical Trials: A Case
> > Studies Approach) is a good read on these sorts of issues and I think
> > there is a more statistical book from the same group of authors.
>
> Thanks for the reference.  My library has it, so will give it a look.
>
> > As far as software is concerned, futility analysis and conditional
> > power are simply standard analyses with made up data and more-or-less
> > justifiable assumptions.
>
> I am also interested if there are good alternatives to conditional
> power for this type of scenario.
>
> > Steve.
> >
> >
> >
> >>
> >> What does this particular Steering Committee think a "futility
> >> analysis" is?  Do they have any particular reference(s)?  What do
> >> you find in your own literature review?
> >>
> >> If it were my problem, I think I'd start with questions like that.
> >> Your comments suggested to me a confounding of technical and
> >> political problems.  The politics suggests the language you need to
> >> use in your response.  Beyond that, I've never heard before of a
> >> "futility analysis", but I think I could do one by just trying to
> >> be clear about the options the Steering Committee might consider
> >> plausible and then comparing them with appropriate simulations --
> >> summarized as confidence intervals, as you suggest.
>
> I did ask REPEATEDLY for guidelines from the steering committee, but
> none came or are likely to come.  In fact, they wanted me to come up
> with the recommendation, which I find entirely inappropriate, but here
> I am.  So, I don't think I'm confounded between techincal and political.

>From what I have seen of the regulatory guidance documents, the SC
should not provide you with any guidelines and the analysis should be
done independent of their input, since their input may be biased in
favor of the new drug. As I note below, the mere fact that the SC is
arguing in favor of continuing the study would suggest the possibility
of a priori bias. This is critical to consider, since there are no
pre-specified stopping rules.

In addition, these should not be done by you in isolation either and the
other clinical members of the DMC/DSMB should materially contribute to
the process. They should be just as clinically competent as any members
of the SC relative to putting forth reasonable assumptions upon which to
base any analyses.

> Basically, they want to stop if there is a low chance of rejecting the
> null hypothesis.  This is often referred to as conditional power or
> stochastic curtailment.  I recently saw a paper by Scott Emerson
> pointing out some problems (interpretation, relation to unconditional
> power).
>
> As far as references, I have used a book by Jennisen and Turnbull in
> the past, but, as I recall, with the exception of stochastic
> curtailment, it assumes the trial was designed with group sequential
> methods.  I have also just found a 1988 Biometrics paper by Lan and
> Wittes on the B-value which I will read.

J&T has a case study with survival analysis as I recall. I don't have it
at hand at the moment.

Another paper that you might find helpful is:

Modifying The Design of Ongoing Trials Without Unblinding
Gould and Shih
Statist. Med., 17, 89 — 100 (1998)

I happened to locate a copy here during a Google search:

http://www2.umdnj.edu/~shihwj/papers/Gould%20and%20Shih%2098.pdf


There is also a Powerpoint presentation by Hung et al that you might
find of interest here:

http://www.fda.gov/cder/Offices/Biostatistics/Hung_etal_6/Hung_etal_6.PPT


BTW, though timing does not help you, there is a new book in process by
Proschan, Lan and Wittes due in June here:

http://www.springer.com/sgw/cda/frontpage/0,11855,4-10134-22-96964889-0,00.html


Two other documents that would be of value here, in a general guideline
sense, are the FDA's draft guidance on DMC's for sponsors (yes, I noted
you are "north of the border"):

http://www.fda.gov/cber/gdlns/clindatmon.htm

and of course ICH E9:

http://www.ich.org/cache/compo/475-272-1.html#E9

Both of which touch on the areas of interim analyses and early stopping.


It seems to me there there is a tension between the sponsor and the SC.

In the former case, the sponsor is looking to make a business decision,
based upon what will clearly be less than perfect data, where
"scientific integrity" is or may be compromised.

On the other hand, the SC is looking to argue for keeping the study
going and that presumably is based upon some a priori insight into the
likelihood of a favorable outcome for the new drug.

If there is to be a mid-course correction in the study (even though
there were no pre-specified stopping rules), there is a risk that the
mere decision to continue the study will somehow bias the future conduct
of it, given the present dynamics and "who knows what" about the
decision making process.

Some of the questions to be considered are what assumptions do you make
in the course of your assessment relative to future data and the need
for adjustments, if any, to the primary hypotheses of the study
(including alpha levels) given the knowledge that becomes available
during the un-planned interim analysis.

At the end of the day, you will need to consider the requirement to make
adjustments in the current study protocol, requiring re-submission and
re-approval by the requisite regulatory authority. It therefore would be
worthwhile to consider proactive communications with the regulatory
contacts for the study and discuss this situation with them to get their
"buy in" on any proposed approaches before taking any further steps.

<snip other content>

HTH,

Marc Schwartz

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html